Mixed-reality endoscope and surgical tools with haptic feedback for integrated virtual-reality visual and haptic surgical simulation

ABSTRACT

An apparatus has a device representing an endoscope, the device being either an endoscope or a dummy endoscope having shape and feel resembling an endoscope, and includes a tracker adapted to operate with a three-dimensional tracking system to track location and orientation of the device in three dimensions in a simulated operating-room environment. The apparatus also has a physical head model comprising hard and soft components, the device representing an endoscope configured to be inserted into the physical head model to provide a haptic feedback of endoscopic surgery.

PRIORITY CLAIM

The present application claims the benefit of priority to U.S.Provisional Patent Application Nos. 62/659,680, 62/659,685, and62/659,672, all of which were filed on 18 Apr. 2018. The entire contentsof each of the forgoing provisional applications are incorporated hereinby reference.

RELATED APPLICATIONS

The present document relates to co-filed applications entitled Systemfor Integrated Virtual-Reality Visual and Haptic Surgical Simulator andSystem for Generating 3D Models for 3D Printing, and for GeneratingVideo for an Integrated Virtual-Reality Visual and Haptic SurgicalSimulator.

FIELD

The present document describes a training, practice, and enhancedoperating environment for surgeons training for, performing, or teachingsurgery. In particular, a training, practice, and performing tele- orvirtual surgical environment is described for Functional EndoscopicSinus Surgery (FESS). This document also highlights segmentation ofcritical structures, including the orbit, brain, cranial nerves, andvessels in augmented-reality and the training and practice environmentincludes visual, auditory, and haptic feedback.

BACKGROUND

Functional endoscopic sinus surgery (FESS) utilizes surgical endoscopesthat allow visualization, magnification and lighting of structures inthe sinuses and nose to perform minimally invasive surgery through thenose. The use of image-guided surgery provides the surgeon withintraoperative landmarks to avoid critical structures in the sinonasalcavity, with the goal of reducing complications into the orbit, brain,cerebrospinal fluid, or major vessels. Although these complications arerare, they can be catastrophic if they occur. FESS is commonly used inthe surgical treatment of chronic sinusitis, the removal of sinonasaltumors, or in access to other craniofacial structures such as theorbital or cranial cavities.

FESS requires rigorous preoperative planning and careful intraoperativedissection of intricate anatomic structures. Due to each individual'sunique anatomy, image-guided surgery is commonly used in complex cases,in which real-time 3-dimensional (3D) tracking systems determinepositions of instruments relative to known skull base anatomy shown onvisual displays. Although image-guided surgery has been shown to behelpful, several studies have shown that complication rates have notsignificantly decreased.

The endoscopes used in FESS are typically rigid endoscopes, providingimage pickup from the surgical field from their distal end. Tools usedin FESS are typically rigid, having a handle, long tubular or rod-shapedshafts, and operative devices at their distal end. These tools areinserted alongside, over, or under the endoscope; once inserted into thesurgical field they are manipulated under visual observation from theendoscope until their distal end and operative devices are positioned asneeded for the operation being performed. When inserting these tools, itis necessary to avoid undue pressure on, or damage to, structures withinthe nasal cavity that are not part of the surgical field—thesestructures are known to. Safe manipulation of these tools and endoscopethrough the obstacle course of turbinates and other structures withinthe nasal cavity and into the surgical field, and use of the tools toperform desired functions, requires practice.

SUMMARY

Our surgical simulation system is a mixed-reality surgical simulatorwith an immersive user experience that may, in some embodiments,incorporate patient-specific anatomy and may be used for preoperativeplanning and practice. The system includes a physical head model, a realor dummy endoscope which can be navigated, a tracking system configuredto track location and angle of the endoscope with 6-degrees-of-freedomin virtual space, trackable instruments either real surgical instrumentsor dummy instruments modeled after real surgical instruments. In someembodiments, new surgical instruments or models thereof may be used. Thetracking system also tracks virtual-reality goggles. A tracking,modeling, and display machine is configured to track a tip of theendoscope within the physical head model and identify correspondinglocations in a CAD model of the physical head and to generate a videostream corresponding to a view of the CAD model from the correspondinglocation in the CAD model. This model allows for: 1) surgical simulationon patient-specific models in virtual reality, 2) the development of anoperating room environment virtually, 3) the use of augmented-reality tohighlight critical structures that can be highlighted through visual orauditory cues, 4) the recording of this virtual surgery on apatient-specific model to then be used as a tracer or guide for traineesperforming live surgery on the specific patient

In an embodiment, an apparatus has a device representing an endoscope,being either an endoscope or a dummy endoscope having shape and feelresembling an endoscope, having an attached tracker adapted to operatewith a three-dimensional tracking system to track location andorientation of the device in three dimensions in a simulatedoperating-room environment. The apparatus also has a physical head modelcomprising hard and soft components, the device representing anendoscope is configured to be inserted into the physical head model toprovide haptic feedback resembling that of using same or similarinstruments and endoscopes in real endoscopic surgery.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a system providing integratedvirtual-reality and haptic feedback from an FESS endoscope to surgicaltrainees or to surgeons preparing for specific cases.

FIG. 2A is a flowchart of a method of training or surgical preparationusing the system of FIG. 1.

FIG. 2B is a detail flowchart of preparing CAD models of hard bony,mucosal, and soft tissues from tomographic radiological image stacks.

FIG. 3 is a block diagram of a system providing integratedvirtual-reality and haptic feedback from an FESS endoscope and surgicaltools to surgical trainees or to surgeons preparing for specific cases.

FIG. 3A is a schematic illustration of critical structures identified inthe radiographic three-dimensional images.

FIG. 3B is an illustration of bony, mucosa, and soft tissue portions ofthe computer-aided design (CAD) model of the head as replicated through3D printing and assembled into the head physical model.

FIG. 4A-4F illustrate of tracker-equipped tools such as may be used withthe endoscope. These include FIG. 4A illustrating an angle-tippedforceps or biter, FIG. 4B illustrating a straight-tipped forceps orbiter, FIG. 4C illustrating an angled-tipped scissors, FIG. 4Dillustrating a straight-tipped scissors, FIG. 4E illustrating astraight-tipped electrocautery, FIG. 4F illustrating a bent-tipped probeor alternatively a bent-tipped cutter.

FIG. 4G illustrates a clamp-on tracker attachment that may be attachedto a 3D printed model of a surgical tool, or to a real surgical tool, totrack the tool in real time.

FIG. 5A is a photograph illustrating a tracker attached to an endoscope.

FIG. 5B illustrates a virtual-reality operating room environment, withdraped patient, endoscope and surgical tool, and endoscopic monitor.

FIG. 6 is a photograph illustrating a hard-plastic physical model

representing bone attached to a tracker to permit easy relative movementanalysis between the physical model and the endoscope tip.

FIG. 7 is a top view photograph of the hard-plastic physical modelrepresenting bone.

FIG. 8 is a photographic view of a tracker attached to a dummyendoscope.

FIG. 9 is a schematic sketch illustrating a tracker attached through aframe to a patient's head.

FIG. 10 is an illustration of a rendered virtual reality view of the CADmodel of the head with superimposed images of specific criticalstructures.

FIG. 11 is an illustration of a rendered virtual reality endoscopic viewas seen from an end of the endoscope with critical structureshighlighted.

FIGS. 12 and 13 illustrate in full and cutaway views an endoscopeinserted into nasal cavities of a physical model of the head, thephysical model having both soft silicone and hard plastic components.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Our surgical simulation system is a mixed-reality surgical simulatorwith an immersive user experience that incorporates patient-specificanatomy.

In an embodiment, a method 200 (FIG. 2) of training or preparation forparticular surgical cases begins with performing a computed tomography(CT) scan 202 using a CT scanner 102 (FIG. 1) to obtainthree-dimensional radiographic imaging of the head 106 of a patient 104in tomographic image stack form; in a particular embodiment thethree-dimensional radiographic imaging of the head 106 includes a stackof tomographic slice images with 0.625-millimeter slices however otherresolutions may be used. In an alternative embodiment, MRI imagers areused in place of CT scanner 102 to image the head and provide a similarstack of tomographic image slices, the stack of tomographic image slicesbeing three-dimensional radiographic imaging.

In a particular embodiment, the CT scan or MRI three-dimensionalradiographic imaging is of the head of a specific patient for which asurgeon wishes a simulated dry run before performing surgery. Inalternative embodiments, the CT scan or MRI three-dimensionalradiographic imaging is, in succession, a CT scan or MRI of a trainingseries of increasing difficulty, including radiographic imaging of headsof patients for which FESS surgery has been performed; with this seriesa beginning trainee surgeon can have a series of VR and physical headmodels prepared with which to learn by practicing basic, moderate, anddifficult surgeries.

The three-dimensional radiographic imaging for a selected head is usedto construct 204, on a model extraction workstation 108, athree-dimensional computer-aided design (CAD) model 110 of the head 106of patient 104, the CAD model 110 includes in separate files a meshmodel of the hard-bony structures of skull, and a mesh model of softtissue structures including mucosa as well as the skin and septalcartilage of nose as illustrated in FIG. 3B. In an embodiment, afterimporting 250 (FIG. 2B) the three dimensional radiographic image stackinto a voxel-based 3-D model, the hard bony structures and soft tissue,including the mucosa, are initially automatically segmented 252, beingdistinguished from each other based at least in part on voxel intensity,and in some embodiments initial segmentation is based on voxelintensity; CT typically shows greater X-ray absorption for calcifiedbony tissue than for soft tissue, while MRI images typically show lesshydrogen for calcified bony tissue than for soft tissue, yet morehydrogen than for voids. Extracted or segmented imaged bony, mucosal,and soft tissue 3D voxel models are processed into mesh models of bony,mucosal and soft tissue structure; mesh model boundaries are generatedwith any inconsistencies (or holes) in the mesh models repaired to formCAD model 110.

Extracting and segmenting imaged bony and soft tissues into 3D meshmodels is performed as illustrated in FIG. 2B. Segmentation of bonytissues into a bony tissue voxel model in an embodiment is done using aregion growing method and threshold process 254, whereas for soft tissueand mucosa voxel models a semi-automatic region growing method is used256; FastGrowCut and the segmentation module in 3-D Slicer functions areused for both the region growing and threshold process and the regiongrowing methods. Skin, muscle, and other soft tissues are segmented frommucosal tissues based upon anatomic landmarks. A marching-cubesurface-mesh reconstruction 258 is then performed on the voxel models toprepare a mesh model of each of the hard bony, mucosal, and softtissues. Manual modification of the threshold and algorithm parameterson a regional basis may be done to avoid over-segmentation,under-segmentation and other artifacts, which may occur with volumeaveraging of bony and soft tissue densities in the head and neck.

The hard-bony tissue mesh model, mucosal tissue mesh model, and softtissue mesh model from the marching cubes reconstructions are thenrepaired 260, first with a surface toolbox mesh-repair module of 3-Dslicer (http://www.slicer.org), and further with Autodesk 3ds Max. In aparticular embodiment, a Lapacian smoothing routine was used to revisemesh models to improve 262 the approximation of curvature without losingvolume.

Both the hard-bony tissue and soft tissue portions of CAD model 110 areconstructed in mesh form using the FastGrowCut Segmentation and Paint(with Editable intensity range for masking) modules in 3-D Slicer andrepaired to eliminate holes with the 3D Slicer surface toolbox. The meshmodels of CAD model 110 are further repaired using Autodesk 3ds Max toreduce the number of vertices for mesh optimization, and to prepare themodel for 3D printing. The generated and repaired mesh models of hardbony tissue, soft tissue, and mucosal tissue form parts of CAD model 110and are exportable in mesh form for use in 3D printers.

In embodiments, CAD model 110 is annotated 206 to build and tag toidentify one or both of a model of a surgical target and models ofcritical structures at risk during FESS surgery or located near to thesurgical target, as illustrated in FIG. 3B. Tagging may be in partmanual and in embodiments may be in part automated using a classifierbased on an appropriate machine-learning algorithm. The annotatedcritical structures may include one or more of the cranial nerves (CN)including the olfactory and optic nerves (CN-1, CN-2) together with theoptic chiasma, anterior and posterior portions of the pituitary gland,the cranial nerves CN-3, CN4, and CN6 that innervate the orbitalmuscles, and the cribriform plate. The critical structures tagged mayalso include blood vessels such as the anterior ethmoidal artery, andinternal carotid artery that can be at risk during endoscopic surgeriesperformed through the nares. The critical structures are identifiedbased upon known anatomic landmarks visible in the three-dimensionalradiological imaging.

In an embodiment, during segmentation soft tissue is identified,including mucosa lining the nasal cavity and paranasal sinuses includingthe inferior, middle, and superior turbinates, maxillary sinuses,anterior ethmoid sinuses, agger nasi, ethmoid bullae, posterior ethmoidsinuses, sphenoid sinus, and frontal sinus.

Neural and arterial structures at risk for damage during surgery wereidentified and segmented separately, these are tagged as criticalstructures so that alarms can be sounded when a virtual surgical toolenters or touches them. These included the anterior ethmoidal artery,internal carotid artery, and cranial nerve II also known as the opticnerve and chiasm. Surface meshes were generated within 3-D Slicer, andexported in wavefront (OBJ) format.

Further, hard tissue is identified based on voxel density including thebone lining the medial orbit known as the lamina papyracea. Bonystructures of the skull identified in this embodiment include theMandible, Maxilla, Sphenoid, Ethmoid, Frontal, and Temporal Bones.

Skin & muscle soft tissues are separated from mucosa based on knownanatomic landmarks.

The bony structures of CAD model 110 are then replicated 208 on a 3Dprinter 112 to prepare a hard-plastic physical model 114 of thosehard-bony structures. In an embodiment, a stereolithographic (SLS) 3-Dprinter based upon photopolymerization of liquid resin is used toprepare hard plastic physical model 114. In a particular embodiment, aFormlabs Form2 (trademark of Formlabs, Inc., Somerville, Mass.) was usedto prepare hard plastic physical model 114 of hard bony parts as definedin CAD model 110. In an alternative embodiment, a fused deposition (FDM)3D printer, such as a Creality Ender 3 (trademark of Shenzhen Creality3D Technology Co., Ltd, Shenzhen, China) or a Zcorp 650 (3D Systems,Rock Hill, South 27 Carolina). was used to prepare hard plastic physicalmodel 114 from polylactic acid (PLA) filament, in other alternativeembodiments hard plastic physical model 114 may be prepared with an FDMprinter using extrudable Nylon or polyethylene terephthalate filamentusing a dual-extruder printer with polystyrene (HIPS) temporarysupporting structures.

3D printer 112 is also used to prepare 210, by 3D printing, a castingmold 116 configured for casting 212 a soft silicone model 118 ofselected soft tissue structures, including skin and septal cartilage ofnose, as described in CAD model 110. In an embodiment, a mold isdirectly printed. In an alternative embodiment, a rigid model of theselected soft tissue structures is printed, this being then used as aform to cast a flexible silicone mold that is in turn used to cast softsilicone model 118 of soft tissue structures. In an alternativeembodiment, soft silicone model 118 is directly printed using a flexibleUV-polymerizable resin in an SLA printer such as the Form2 printer

3D printer 112 is also used to prepare 211 a casting mold 117 configuredfor casting 213 a soft silicone model 119 of selected mucosalstructures, such as line the interior of nasal cavity and sinuses, asdescribed in CAD model 110. Once cast 213, the soft silicone mucosalmodel 119 is mounted 215 to the hard-plastic physical model 114. In analternative embodiment, model 119 of mucosal structures has beendirectly 3D printed using an SLS-type 3D printer such as a Form2 printerand flexible, UV-curable, resin.

With reference to FIG. 3B, once the soft silicone model 118 of softtissue structures is cast, 212, and after mounting 215 the mucosal model119 to the hard plastic bony tissue model, the soft silicone model 118of soft tissue structures is mounted 214 to the hard plastic physicalmodel 114 of bony tissues to create the head physical model 115, thehead physical model 115 including hard plastic physical model 114, softsilicone model 119 of mucosal structures, and soft silicone model 118 ofsoft tissue structures.

CAD model 110 is also loaded 216 into a mechanical modeling and trackingmachine 122 equipped with tracking receivers 124, 126. Trackingreceivers 124, 126 are configured to track 218 location and orientationin three-dimensional space of a tracker 128 that is attached to a dummyendoscope 130, in a particular embodiment, tracking receivers 124, 126and tracker 128 are HTC Vive (HTC, New Taipei City, Taiwan) trackers andthe virtual reality goggles are an HTC Vive headset; other virtualreality goggles and trackers may be used. In an embodiment, headphysical model 115 is at a known location, in other embodiments, hardplastic physical model 114 is attached to another tracker 150 through ashort steel rod 152 as illustrated in FIG. 6. Also attached to dummyendoscope 130 is endoscope handle 132 that may include operativecontrols. In a particular embodiment, operative controls on endoscopehandle 132 include camera angle selection buttons. In an alternativeembodiment, a real endoscope may be used in place of the dummyendoscope. For purposes of this document, a device representing anendoscope may be either a dummy endoscope or a real endoscope.

The mechanical modeling and tracking machine 122 then uses the locationand orientation of the tracker 128 on the endoscope 130 to determine 220a location and orientation of endoscope head 134 in the head physicalmodel, which is in turn aligned and registered to a virtual head asmodeled by CAD model 110 executing on modeling and tracking machine 122,the CAD model 110 being derived from the 3D image stack determined fromMRI or CT scans. Since the head physical model is registered to the CADmodel 110, each location of endoscope head 134 in the head physicalmodel corresponds to a location in the CAD model 110.

Interaction of the device representing an endoscope with the headphysical model as the device is inserted into the model provides tactileor haptic feedback to a surgeon or trainee that resembles tactile orhaptic feedback as the surgeon or trainee inserts a real endoscope intoa patient's real head.

An endoscope alone, however, is useful to visually inspect internalsurfaces within the nasal cavity but cannot by itself perform FESSsurgery. To perform surgery, additional surgical tools are inserted intoa patient's head along with the endoscope.

To provide simulated tactile or haptic feedback to a surgeon or surgicaltrainee of manipulation of surgical tools in a head as well as feedbackof manipulating the endoscope, in embodiments one or more devicesresembling surgical tools are provided (FIG. 4). These tools may includeany combination of an angle-tipped forceps or biter 460 illustrated inFIG. 4A, a straight-tipped forceps or biter 462 FIG. 4B, anangled-tipped scissors 464 FIG. 4C, a straight-tipped scissors 466 FIG.4D, a straight-tipped electrocautery 468 FIG. 4E, a bent-tipped probe oralternatively a bent-tipped cutter 470 FIG. 4F, a bent-tippedelectrocautery (not shown), a bent-tipped or straight-tipped drills (notshown), straight or bent suction tubes (not shown), microdebriders (notshown), straight tipped probes and cutters (not shown), and other toolsas known in the art of FESS surgery; the tool may in an embodiment be anew or experimental tool of unique shape. Each device resemblingsurgical tools may be a 3-D print of a tool with an embedded 3-D tracker402, 404, 406, 408, 410, 412, or may be a real surgical tool with aclamp-on 3-D tracker as illustrated in FIG. 4G. The clamp-on 3-D trackerhas a 3-D tracking device 420 and clamp 422 and is configured to mountwith a setscrew 424 to a shaft or body 426 of a surgical tool or 3-Dmodel of a tool. Each tool has a corresponding 3D mesh model used in thegaming engine of the modeling and display machine to track position ofthe tool tip and to derive an image of the tool tip when the tool tip isin a field of view of the simulated endoscope tip. In an embodiment, thedevices resembling surgical tools may be equipped with short-rangeradio-activated vibrators to provide haptic feedback resembling that ofan operating drill or to provide alarms generated when tool tipsapproach critical structures.

Tools used in FESS, such as forceps, biters, and scissors, often have along, narrow, shaft 450, 452 configured to fit through the nares intothe nasal cavity, they also have a handle 440, 442, 444 that allows theuser to control their angle of orientation within the nasal cavity.These tools operate when an operating lever 430, 432, 434 is pressed,the operating lever being coupled through an operating rod that istypically disposed within the shaft 450, 452. Simple cutters and probes,as illustrated in FIG. 4F, do, however, lack an operating lever. Forincreased realism, operating levers of devices resembling a surgicaltool may be instrumented with sensors configured to sense pressure onthe operating lever, and transmit sensed pressure to the video modelingand display machine 136.

FIG. 4G illustrates a clamp-on tracker attachment that may be attachedto a 3D printed model of a surgical tool, or to a real surgical tool, totrack the tool in real time.

A computer model of each tracker-equipped surgical tool 460, 462, 464,466, 468, 470 is incorporated into the mechanical model and trackingmachine 122 and video model and display machine 136. The mechanicalmodel and tracking machine 122 uses information received throughmultiple tracking receivers 124, 126 to determine position andorientation of both the tracker 128 on the endoscope 130 (FIG. 3) andtracker 160 on the tracker-equipped tool 162 to determine position andorientation of the tip 134 of the endoscope and operating portion 164 ofthe tool.

Meanwhile, a video modeling and display machine 136 executes a videogame engine, in an embodiment the video game engine is the Unity GameEngine, in a particular embodiment Unity Engine V2017.3, (Trademark ofUnity Technologies, San Francisco, Calif.) was used, the video modelingand display machine 136 also executes the CAD model 110 of the head.Together the mechanical modeling and tracking machine and video modelingand display machine form a tracking, modeling, and display machine. Inan alternative embodiment, modeling and tracking machine 122 and videomodeling and display machine 136 are combined within a single tracking,modeling, and display machine executing a plurality of modules.

Video modeling and display machine 136 executing a video gaming engine138 determines objects represented in CAD model 110 that are in view ofthe endoscope head 134, including anatomy of the head, at one of threeselectable endoscope viewing angles, and renders 222 the objects in viewof the endoscope head 134 into a video image. The objects represented inCAD model 110 may include models of foreign objects or tumors 166 uponwhich surgery is to be conducted. The gaming engine 138 also determineswhether a tip 164 of any device resembling a surgical tool 162 is in afield of view of the endoscope as oriented, and renders that into thevideo image. In an embodiment the game engine is the Unity Game Enginev2017.3 (Unity Technologies). The present system is adapted to renderobjects in view of straight as well as angled endoscopes with accuratefield of view. The game engine includes capability of photo-realisticrendering in real-time with dynamic lighting sources and shadows, in anembodiment the dynamic lighting source is chosen to correspond to alighting fiber of a real endoscope so rendered images strongly resembleimages seen through an endoscope camera during live surgeries. Thisvideo image represents a view corresponding to a view through anendoscope at a corresponding position in the patient's head 106. Thevideo image corresponding to a view through the endoscope tip may thenbe tagged 224 with indications of critical structures and presented 226to a trainee or operating surgeon through virtual reality goggles 140 asif on an endoscope monitor with images of other objects in a virtualoperating room. Virtual reality goggles 140 are also equipped with atracker 146.

Mechanical modeling and tracking machine 122 compares computed locationsof both the endoscope tip 134 and tool tip 164 to locations of criticalstructures as flagged in model 110, and provides alerts when either tip134, 164 is positioned to damage those critical structures. Thesecritical structures include the orbits, cribriform plate, cavernoussinus, and multiple cranial fossae of the skull; when the video modeland game machine 136 detects entry of a simulated surgical tool tip intoor against one of these critical structures, the video model and gamemachine sounds an audible alarm or displays a visual alarm; in someembodiments a vibrator is used to provide a haptic alarm. In analternative embodiment, alarms are generated upon a simulated surgicaltool tip approaching one of these critical structures that have beentagged in the mucosal mesh model.

The system includes, within video model and game machine 136, a virtualreality model of a virtual operating room, including 3-D models of muchcommon operating-room equipment such as an operating table, instrumenttray, electrocautery machine, endoscope illuminator/camera controller,and endoscope monitor.

In operation, a trainee or operating surgeon puts on virtual realitygoggles 140 then picks up and manipulates the endoscope 130 to insertendoscope head 134 into nares 142 of head physical model 115 into nasalcavity 144 of head physical model 115; the trainee or surgeon may alsoinsert one or more tools 162 through the nares 142 into nasal cavity144. While the surgeon is inserting the endoscope and tools, tracker 146tracks location and angle of virtual reality goggles 140 to permitsynthesis in video model and game machine 136 of a video streamincorporating a view of the virtual operating room with a virtualpatient having head aligned and registered with a physical location ofphysical head model 115, and draped as typical for FESS surgery. In anembodiment, the view of the virtual operating room includes an image ofan endoscope aligned and positioned according to tracked position andorientation of endoscope 130. The virtual operating room includes avirtual operating room monitor providing the virtual reality renderedvideo image as viewed from the endoscope tip, potentially including animage of the surgical tool tip 504 as well as an image of tumor to beresected 506, permitting the trainee or operating surgeon to view therendered video image by aiming his or her head, and virtual realitygoggles 140, at the virtual operating room monitor 502, as illustratedin FIG. 5B. Also visible in the VR goggle display may be, depending onVR goggle position and orientation, the simulated head 508 of properlydraped patient 510, endoscope 512, and surgical tool 514

In an embodiment, the tracking and modeling machine 122 tracks positionof the endoscope head 134 in physical model 115 and provides alerts whenendoscope head 134 approaches locations corresponding to tagged criticalstructures in CAD model 110. In an embodiment these alerts are providedas aural alerts and as visual alerts by superimposing warnings andimages of critical structures on the virtual endoscope images presentedon the virtual operating room monitor thereby simulating an alternativeembodiment that presents visual warnings on actual endoscope imagesduring live surgeries.

While the trainee or surgeon manipulates the endoscope and surgical toolor tools, mechanical interactions of endoscope 130 and endoscope head134 with the head physical model 115 provide tactile, or haptic,feedback to the trainee or operating surgeon, the tactile feedbackgreatly resembling tactile feedback felt during actual surgeries onsinuses, pituitary, and other organs accessible to endoscope 130 throughnares 142. Tactile and haptic feedback is inherent to using 3D printeddummy endoscopes and other tools in the shape of real surgical tools,and having a trackable patient skull with anatomic features with whichthe endoscope and other surgical tools physically interact. One aspectof tactile feedback is the feel of the endoscope and its controller, andwhen present the surgical tools, in the trainee's hands each with 6 fulldegrees of freedom, providing a proprioceptively authentic feel in aroom-scale immersive virtual-reality environment.

In embodiments, dummy endoscopes and dummy surgical tools are 3D printedwith FDM printers.

In an alternative embodiment tactile feedback is enhanced with avibratory mechanism within the dummy endoscope or other dummy tools tosimulate a surgical drill, suction probe or suction cautery such as maybe used during actual surgeries.

Similarly, the virtual reality rendered video image presented on thevirtual operating room monitor with virtual reality goggles 140 providesvisual feedback like visual images seen by a trainee or operatingsurgeon while performing similar operations. The position and angle ofthe VR goggles are tracked and the simulated OR environment is displayedthrough the VR goggle with position and size of the simulated endoscopemonitor dependent on angle and position of the VR goggle. In this way,movement of the trainee or operating surgeon's head provides realisticmovement of stationary objects in his field of view like the simulatedendoscope monitor while he is wearing the VR goggles. Both the headphysical model 115 and virtual reality rendered video based on CAD model110 are patient-specific since CAD model 110 is derived from thethree-dimensional radiographic images of a specific patient's head 106.

In an embodiment, dummy endoscope 130 has a lumen and operative toolscan be inserted through that lumen, in particular embodiments thesetools may include drills for penetrating through bone into sinuses orthrough bone to reach a pituitary gland; these tools can also penetratethrough hard plastic of physical model 114 during practice procedures.

In an alternative embodiment, for use in live surgeries, a frame 304 isattached to the patient's head 106, and a tracker 306 is positioned onthe frame. The patient's head is registered to the tracking system withthe CAD model 110 aligned to the patient's head 106. In this embodiment,the tracking and modeling machine 122 tracks position of the endoscopehead and provides alerts when tracked endoscope head 134 approachestagged critical structures as identified in the CAD model 110, in anembodiment these alerts are provided as aural alerts and as visualalerts by superimposing warnings and images of critical structures onimages obtained through an endoscope camera viewing the patient's nasalcavity and sinuses from endoscope head 134.

In an alternative embodiment, critical structures may be highlighted anddisplayed as illustrated in FIG. 10 as structures shown with referenceto the head and FIG. 11 as highlighted structures in an endoscope view.FIG. 10 illustrates critical structures viewed in projectionsuperimposed on the CAD model 110.

In an alternative embodiment, the entire motion of the endoscope andoperative tools is recorded by the operating surgeon and thentransmitted to another site to provide a tracing of the surgery to bethen mirrored by a second surgeon performing live surgery(tele-surgery), or repeated by trainees to provide repetitive guidedtraining.

In an alternative embodiment, positions of head physical model orpatient head, and endoscope as detected by the trackers are recordedthroughout a practice or real surgical procedure. In an embodiment, ascore is produced based on time to perform a predetermined task withpenalties applied for approach of simulated tool tip to simulatedcritical structures; in embodiments motion tracking of tool andendoscope is used to determine economy of movement and the score is alsobased on economy of movement. In a particular embodiment, amachine-learning procedure is trained on beginning and experiencedsurgeons and motion tracking to provide personalized feedback to traineesurgeons and score users on their relative level in performing surgery.This feedback could be used to advance users from a beginner to expertlevel, or evaluate the level of surgeons in the community. Relativemotions of endoscope and instrument to head as recorded are thenanalyzed using the 3D CAD model and critical structures tagged in theCAD model to provide feedback to the trainee surgeon. Such analysis mayinclude indications of safer or faster ways the procedure could beperformed, or be used to evaluate surgeons already performing surgery.For purposes of this document, derivation of the score and its use intraining surgeons by giving real-time feedback to users, either byaltering the level of difficulty of the simulation, providingvisual/auditory/haptic feedback and cues to assist in surgery, andprovide objective feedback or score on the simulation is known as thevirtual coach. This could be used to evaluate proficiency duringtraining, as well as provide a method of continued certification forpracticing surgeons.

In an alternate embodiment, trackers are coupled to a real endoscope andreal surgical tools, and a tracker on a frame is clamped to the samepatient's head as used to generate the CT or MRI radiologicaltomographic image stack from which the CAD model was derived. Thephysical head model is not used in this alternate embodiment, the CADmodel is registered to the patient's head. The modeling and displaymachine tracks locations of the endoscope and surgical tools tips in theCAD model—corresponding to positions in the patient's head—and generatesvisual or aural alarms when these tips approach critical structurestagged in the CAD model. These alarms serve to assist surgeons inavoiding damage to those critical structures.

For purposes of this document, the term “resilient polymer” shallinclude rubberlike polymeric materials, including polymerized Fromlabselastic resin, resilient silicones and some soft carbon-based syntheticrubbers and flexible plastics like molded latex and sorbothane, adaptedto being formed into flexible reproductions of human soft tissue such asskin and muscle and having Shore-A hardness no greater than 85. The term“hard plastic” shall include polymeric materials significantly harderthan resilient polymers as defined above, including most acrylonitrilebutadiene styrene (ABS), high impact polystyrene (HIPS), and polylacticacid (PLA) 3D printer filaments, and polymerized Formlabsstandard-hardness grey resin.

Experimentally, Vive trackers were reliably tracked by Vive lighthousebase stations to less than a centimeter, updating the position of thetools and user in the virtual environment without detectable latency.The endoscope could register correctly the modeled danger-zones withaudio and visual cues time-synchronously. This framework provides acost-effective methodology for high-fidelity surgical trainingsimulation with haptic feedback. Through virtual simulation,personalized training programs could be developed for trainees that areadaptive and scalable on any range of difficulty and complexity.Proposed approaches to VR can be extended to the telemedicine world, inwhich surgeons operating in remote locations can be assisted by theexperts aiding from tertiary care centers. State-of-the-art surgicalnavigation systems such as the system herein described provide reliableoptical and electromagnetic-based tracking with accuracy withinpotentially 2 mm. These navigation workstations confirm anatomiclocation but do not reduce the risk of surgical complications down to0%. Additional features from our technology could be translatable todevelop AR-based navigation, which can further improve safety in theoperating room.

Combinations of Features

The features herein described may be combined into a functional surgicalsimulation system and environment in many ways. Among ways anticipatedby the inventors that these features can be combined in variousembodiments are:

A multimode VR apparatus designated A including an endoscope deviceadapted to represent an endoscope, the endoscope device selected from anendoscope and a dummy endoscope having shape and feel resembling that ofan endoscope; a wireless tracker adapted to operate with athree-dimensional tracking system to track location and orientation ofthe endoscope device in three dimensions in a simulated operating roomenvironment; and a video modeling and display machine configured with acomputer-aided design (CAD) model of a head and adapted to provide asimulated head environment, providing a simulated endoscope view. Theapparatus also includes a physical head model comprising hard and softphysical components, the endoscope device being configured to beinserted into the physical head model to provide a tactilerepresentation of manipulation of an endoscope in a head to a personhandling the endo-scope device.

An apparatus designated AA including the multimode VR apparatusdesignated A wherein the video modeling and display machine comprises agaming engine adapted simulate endoscope view of the simulated headenvironment

An apparatus designated AB including the apparatus designated A or AAwherein the physical head model comprises a wireless tracker, and wherethe computer-aided design (CAD) model of a head is registered to atracked position of the physical head model.

An apparatus designated AC including the apparatus designated A, AA, orAB wherein the physical head model comprises a hard-plastic portionprepared by 3D printing representative of bony tissue and a resilientpolymer portion representative of skin.

An apparatus designated AD including the apparatus designated A, AA, ABor AC and including a surgical tool device having shape and feelresembling that of a surgical tool adapted for functional endoscopicsinus surgery (FESS) selected from the group consisting of forceps,biting forceps, scissors, a probe, and an electrocautery, the tooldevice further comprising a wireless tracker adapted to operate with thethree-dimensional tracking system to track location and orientation ofthe tool device in three dimensions in the simulated head environment.

An apparatus designated AE including the apparatus designated AD whereinthe simulated endoscope view includes a simulated view of the tooldevice.

An apparatus designated AF including the apparatus designated A, AA, AB,AC, AD or AE wherein the video modeling and display machine is furtherconfigured to provide a simulated operating room (OR) environment withthe simulated endoscope view displayed on a simulated endoscope monitor.

An apparatus designated AFA including the apparatus designated A, AA,AB, AC, AD, AE, or AF wherein the tool device resembles a surgical toolselected from the group consisting of forceps, biting forceps, scissors,a probe, a drill, and an electrocautery.

An apparatus designated AG including the apparatus designated A, AA, AB,AC, AD, AE, or AF further including a virtual-reality (VR) goggleequipped with a wireless tracker, and wherein the simulated ORenvironment is displayed through the VR goggle with position and size ofthe simulated endoscope monitor on the VR goggle display dependent onangle and position of the VR goggle.

A method designated B of preparing a physical model of a head andendoscope for surgical simulation includes importing into a workstationa radiological tomographic image stack of a head; segmenting theradiological tomographic image stack into hard tissue, soft tissue, andmucosal voxel models based at least in part on voxel intensity; andgrowing hard tissue, mucosal, and soft tissue regions in the hardtissue, soft tissue, and mucosal voxel models. The method continues withconverting the hard tissue, soft tissue, and mucosal models into a hardtissue mesh model, a soft tissue mesh model, and a mucosal mesh model;repairing the mesh models; exporting the mesh models from theworkstation and using a 3D printer and the hard tis-sue mesh model toprint a physical hard tissue model; preparing a physical mucosal tissuemodel from the mucosal mesh model; and mounting the physical mucosaltissue model to the physical hard tissue model. The method also includespreparing a physical soft tissue model from the soft tissue mesh model;and mounting the physical soft tissue model to the physical hard tissuemodel to form a physical head model. The method also includes loadingthe mesh models into a display system adapted to render images ofsurfaces of the mesh models as viewed from an endoscope; mounting atracker to the physical head model; and preparing an endoscope devicewith a tracker.

A method designated BA including the method designated B and including:tracking the endoscope device to determine a tracked endoscope positionand orientation; determining a location and orientation of a tip of theendoscope device from the tracked endoscope position and orientation,the position of the tip of the endoscope device being within thephysical head model; rendering images of surfaces of the mesh models asviewed from the tip of the endo-scope device; and displaying the imagesof surfaces of the mesh models.

A method designated BB including the method designated B or BA furtherincludes mounting a tracker to a device representing a surgical tool;tracking the device representing a surgical tool to determine a locationof a tip of the device representing a surgical tool; determining whenthe surgical tool is in view of the tip of the endoscope device; andwhen the surgical tool is in view of the tip of the endoscope device,rendering an image of a surgical tool as viewed from the tip of theendoscope device.

A method designated BC including the method designated B, BA, or BBfurther includes: tracking location and orientation of the physical headmodel; and registering the mucosal mesh model to the location andorientation of the physical head model.

A method designated BD including the method designated B, BA, BB, or BCwhere the rendering images of surfaces of the mesh models is performedwith a 3D gaming engine.

A method designated BE including the method designated B, BA, BB, BC, orBD wherein the preparing a physical mucosal model is performed bycasting using a mold that has been prepared from the mucosal mesh modelby a method comprising 3D printing.

A method designated BF including the method designated B, BA, BB, BC,BD, or BE further includes identifying critical anatomic structuresimaged in the radiological tomographic image stack and tagging thosecritical structures in a model of the mesh models.

A method designated BG includes the method designated B, BA, BB, BC, BD,BE, or BF and further includes generating alarms upon approach of a tipof the device representing a surgical tool to a critical structuretagged in the mesh model.

An endoscopic surgical simulation system designated C includes aphysical head model; a tracking system configured to track location andangle of a device representing an endoscope and a device representing asurgical tool; a computer-aided design (CAD) model in a modeling, anddisplay machine, the CAD model registered to a location of the physicalhead model and comprising CAD representations of structurescorresponding to structures of the physical head model; with themodeling, and display machine being configured to track the devicerepresenting an endoscope and determine a location of a tip of thedevice representing an endoscope within a nasal cavity of the physicalhead model, and to determine a field of view of an endoscope located atthe location of the tip of the device representing an endoscope. Themodeling, and display machine is configured to track the devicerepresenting a surgical tool and determine a location of a tip of thedevice representing a surgical tool within the nasal cavity of thephysical head model; and the modeling and display machine is configuredto generate a video stream corresponding to a view of structuresrepresented by the CAD model within the field of view. The modeling anddisplay machine is also configured to superimpose on the video stream animage corresponding to a tip of a surgical tool when the location of atip of the device representing a surgical tool is in a field of view ofview.

An endoscopic surgical simulation system designated CA including theendoscopic surgical simulation system designated C wherein the CAD modelincludes models of a plurality of structures tagged as criticalstructures.

An endoscopic surgical simulation system designated CB including theendoscopic surgical simulation system designated C or CA furtherincluding a tracker coupled to the physical head model, and wherein theCAD model is registered to a location of the physical head model.

An endoscopic surgical simulation system designated CBA including theendoscopic surgical simulation system designated C, CA, or CB whereinthe physical head model and CAD model are derived from computedtomography (CT) or magnetic resonance imaging (MRI) scans of aparticular patient, the system configured for preoperative planning andpractice for that particular patient.

An endoscopic surgical simulation system designated CBB including theendoscopic surgical simulation system designated C, CA, CB wherein thereis a first physical head model and CAD model configured for a firsttask, and a second physical head model and CAD model configured for asecond task, the second task of greater difficulty than the first task.

An endoscopic surgical simulation system designated CC including theendoscopic surgical simulation system designated C, CA, CB, CBA, or CBBwherein the modeling and display machine is configured to generatealarms upon approach of the location of a tip of the device representinga surgical tool to a structure tagged as a critical structure.

An endoscopic surgical simulation system designated CD including theendoscopic surgical simulation system designated C, CA, CB, CC, or CBAfurther including a model extraction workstation configured to extractthree-dimensional mesh models from computed tomography (CT) or magneticresonance imaging (MRI) radiographic images, and wherein the physicalhead model is generated by a method comprising 3D printing of extractedthree-dimensional mesh models.

An endoscopic surgical simulation system designated CD including theendoscopic surgical simulation system designated C, CA, CB, CBA, CBB, orCC further including virtual reality (VR) goggles, the VR gogglesequipped with a tracker.

An endoscopic surgical simulation system designated CE including theendoscopic surgical simulation system designated CD wherein the videostream corresponding to a view of structures represented by the CADmodel within the field of view is displayed upon a display of the VRgoggles.

An endoscopic surgical simulation system designated CF including theendoscopic surgical simulation system designated CE where the videosteam corresponding to a view of structures represented by the CAD modelis displayed on the VR goggles at a position dependent on location andorientation of the VR goggles.

It should thus be noted that the matter contained in the abovedescription or shown in the accompanying drawings should be interpretedas illustrative and not in a limiting sense. The following claims areintended to cover all generic and specific features described herein, aswell as all statements of the scope of the present method and system,which, as a matter of language, might be said to fall therebetween.

What is claimed is:
 1. A method of preparing a physical model of a headand endoscope for surgical simulation comprising: importing into aworkstation a radiological tomographic image stack of a head; segmentingthe radiological tomographic image stack into hard tissue, soft tissue,and mucosal voxel models based at least in part on voxel intensity;growing hard tissue, mucosal, and soft tissue regions in the hardtissue, soft tissue, and mucosal voxel models; converting the hardtissue, soft tissue, and mucosal models into a hard tissue mesh model, asoft tissue mesh model, and a mucosal mesh model; repairing the meshmodels; exporting the mesh models from the workstation and using atleast a 3D printer and the hard tissue mesh model to generate a physicalhead model; loading the mesh models into a display system adapted torender images of surfaces of the mesh models as viewed from anendoscope; mounting a tracker to the physical head model; and preparingan endoscope device with a tracker.
 2. The method of claim 1 whereingenerating a physical head model is performed by printing a hard plasticphysical hard tissue model from the hard tissue mesh model; preparing aresilient polymer physical mucosal tissue model from the mucosal meshmodel; mounting the physical mucosal tissue model to the physical hardtissue model; preparing a resilient polymer physical soft tissue modelfrom the soft tissue mesh model; and mounting the physical soft tissuemodel to the physical hard tissue model to form the physical head model.3. The method of claim 1 further comprising: tracking the endoscopedevice to determine a tracked endoscope position and orientation;determining a location and orientation of a tip of the endoscope devicefrom the tracked endoscope position and orientation, the position of thetip of the endoscope device being within the physical head model;rendering images of surfaces of the mesh models as viewed from the tipof the endoscope device; displaying the images of surfaces of the meshmodels.
 4. The method of claim 3 further comprising: mounting a trackerto a device representing a surgical tool; tracking the devicerepresenting a surgical tool to determine a location of a tip of thedevice representing a surgical tool; determining when the surgical toolis in view of the tip of the endoscope device; and when the surgicaltool is in view of the tip of the endoscope device, rendering an imageof a surgical tool as viewed from the tip of the endoscope device. 5.The method of claim 4 further comprising: tracking location andorientation of the physical head model; and registering the mucosal meshmodel to the location and orientation of the physical head model.
 6. Themethod of claim 1 where the rendering images of surfaces of the meshmodels is performed with a 3D gaming engine.
 7. The method of claim 2wherein the preparing a physical mucosal model is performed by castingusing a mold that has been prepared from the mucosal mesh model by amethod comprising 3D printing.
 8. The method of claim 1, 2, 3, 4, 5, 6,or 7 further comprising identifying critical anatomic structures imagedin the radiological tomographic image stack and tagging those criticalstructures in a model of the mesh models.
 9. The method of claim 8further comprising generating alarms upon approach of a tip of thedevice representing a surgical tool to a critical structure tagged inthe mesh model.
 10. The method of claim 9 further comprising generatinga score based upon at least time of a trainee completing a task andapproach of simulated endoscope tip and simulated tool tip to thecritical structures tagged in the model.
 11. The method of claim 8wherein the physical head model and the endoscope device are configuredto physically interact upon insertion of the endoscope device into anasal cavity of the physical head model and thereby providing hapticfeedback to a user, the haptic feedback approximating haptic feedbackwhen the user inserts a real endoscope into a nasal cavity of a realhuman head.